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ABSTRACT 

Bow data gathered through item response theory (IRT) 
analysis can be used to give immediate diagnostic information without 
a computer is explained. The Kidmap, developed by B. D. Wright and 
others (1980), provides clear information in a visual manner. Its 
major problem lies in the need for a computer and suitable software 
to construct a Kidmap. The solution lies in the proposed diagnostic 
map (DIAMAP) • Once items have been calibrated and a raw score to 
logits equivalence established, items that would normally be placed 
in correct and incorrect columns on a Kidmap are put on either side 
of the central raw score scale. Items answered correctly on the 
"correct * side are circled, and incorrectly answered items on the 
"incorrect side" are circled. The raw score is marked on the central 
column and interpretation proceeds as for a Kidmap. An analogous 
version is also created for rating scale analysis; this version can 
give an immediate understanding of the measured trait for every 
respondent, baaed solely on raw score. This brief outline of the 
construction and use of DIAMAPs illustrates that good reporting 
methods can make IRT analysis more pertinent to the lay user. Eight 
figures illustrate Kidmaps and DIAMAPs. A nine-item reference list is 
included. (SLD) 
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fjtj The purpose of this paper is to explain how data prepared through Item 

Response Theory analysis can be used to give immediate diagnostic information 
without the need for a computer. 

In his farewell 'A Note from the Chair 7 David Andrich (1992), with reference 
to Item Response Th . y analysis, asks 

"Why hat ro much good work done over so many years by so 
many dedicated, intelligent people been so ignored by the 
leaders of the educational community?" 

(page 191) 

The reasons for this state of affairs may be obscure but I would like to suggest 
a possible cure. Better reporting of analysis! It is my belief that first class reporting 
of results of analyses would promote a wider use of IRT. If IRT methods are more 
informative, then they will gain more favour within the educational community and 
a more useful tool is more likely to be used. 

The function of analy. c >s as is said frequently at these meetings, is to let the 
data speak for themselves, but it is most often the case that the analysis, no matter 
how psychometrically sound, does not give the data a clear voice. This is true 
whether the analysis is traditional or Item Response Theory based. 

The exception to this is of course where Kidmaps (Wright, Mead and Ludlow; 
1980) have been produced by such analysis programs as TITAN. The Kidmap 
provides in a visual manner, clear information , information that otherwise would 
be hidden in the welter of tables and figures produced by the anrlysis. 

The advantages of the Kidmap over the usual stanines, percentiles and 
whatnots is clear once one has used it. A classic case of a Kidmap giving 
immediately useful diagnostic information is shown in Figures 1 and 2. 

- Figure 1 gives the first picture or map of Tony's knowledge in basic arithmetic. 

There are six items whose difficulty would lead one to expect Tony to get them 
>» correct (D17, D13, E04, C14, C07, and E14). As has been noted by his teacher there is 

a major problem with subtraction. 

o BEST COPY AVAILABLE 



Figure 1 : Kidmap for Tony in basic arithmetic (Masters et al; 1987). 
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The second of Tony's Kidmaps, constructed a year later, shows that his subtraction 
problem has not been corrected and is now considered serious. 
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The guidance for teachers provided by the Kidmap is a major advantage of 
over traditional reporting methods, where often one cannot see the wood for the 
trees. However even the Kidmap has disadvantages! The major problem lies with 
the need for a computer and suitable software to construct such Kidmaps. The 
solution lies in what I have called a diagnostic map or DIAMAP. 

The idea of the PIAMAP is quite simple. Once items have been calibrated and 
a raw score to logits equivalence established, a modified Kidmap can be constructed. 
The modification is to take the items from the Kidmap, which normally would be 
placed in the correct and incorrect columns as appropriate, and put them all in both 
columns at either side of the central raw score scale. This gives a DIAMAP like the 
one in Figure 3. 




Figured Sample DIAMAP (Doig;1990). 



The next step in the construction of the DIAMAP requires one to circle all 
those questions which have been answered correctly on the 'correct' (left hand side) 
of the DIAMAP and to circle only the incorrectly answered questions on the 
'incorrecf (right hand side) of the DIAMAP. Finally, the raw score is marked on the 
central score column. The interpretation of the DIAMAP then proceeds as for a 
Kidmap. An example of a completed DIAMAP from the Diagnostic Mathematics 
Profiles (basic arithmetic) is shown in Rgure 4. 




Figure 4: Completed DIAMAP from the Diagnostic Mathematics Profiles 
(Doig; 1990). 



6 



» 



To further assist in interpretation, a DIAMAP can be divided horizontally into 
'bands' or levels' for which descriptions are provided. This process is well 
illustrated by Figure 5, which is the DIAMAP for the Australian Language 
Certificate in French Listening (Zammit; 1991). 
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Level 



At Uus level a student can 
understand a range of 
everyday words and 
phrases, * able to follow 
directions and identify 
several items of 
information in a short 

conversation. 



A student at this level can 
understand a range of 
everyday words, simple 
requests and greetings and 
can listen for details. 



I 

A typical student at this 
level can understand some 
everyday words, simple 
requests and greetings as 
well as identify single 
pieces of information. 



Preliminary 

A student at this level has 
yet to master the 
introductory elements of 
French. 



Figure 5: Completed DIAMAP from the Australian Language Certificate in French 
(Listening) (Zammit; 1991). 
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DI AMAPs need not be identical in either their lay-out or in the manner in 
which they are marked. For example, the DIAMAP for the 'Beats and Bars' music 
assessment shown in Figure 6 uses a 'triple' version to show the three sub-tests at 
one glance (Brown and Doig; in preparation). 
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Beats and Bars DIAMAP 
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Comments . 



Outline of Test Item Topics 

Repeat Markings 

Draw repeat marks 

Rewrite bars using repeats. 

Rewrite bar order, interpreting repeats 

Count the notes in a repeated passage. 

Write full order of bars that use time bars. 

Count the repetition of a bar in music with time bars. 

Time signatures 

Draw time signature in position on stave 
Write a 4/4 rhythm. 

Describe function of the top number in time signature. 
Describe function of the bottom number in time signature. 
Identify the correct time signature of music. 
Complete a bar with notes/rests according to the time signature. 

Beats within a bar 

Indicate beats in bars of music. 

Indicate notes not on the beat 

Write the beats under notes in the bars. 

Mark certain beats in the music 
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Figure 6: DIAMAP for Beats and Bars Music Assessment (Brown and Doig; in 
preparation). 
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The marking for this music DIAMAP uses circles to indicate correctly 
answered questions and boxes to identify incorrectly answered questions. This 
helps to minimise the number of items on the DIAMAP when a lot information 
needs to be included on the page. The Tests of Basic Skills in Language DIAMAP of 
Figure 7 shows another variant. This time it was thought necessary to show items 
ommitted, so these were enclosed in boxes to distinguish them from the other 
'incorrect* items. 
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Figure 7: Completed DIAMAP from the Test of Basic Skills (Lokan; 1991). 



A further step in the development of the DIAMAP is to create the analogous 
version for rating scale analysis. A typical problem is that traditional methods leave 
one without a 'feel' for the individual's results. Fbr example, in attitude 
measurement, a score of 60 on a 30 item, four category (1, 2, 3, or 4) questionnaire 
gives no indication of the respondent's attitude to particular questions, or sub-sets of 
questions. However the use of Rating Scale analysis can improve the information 
gained from such questionnaires as shown by Masters and Hyde in their study of 
attitude to school (Masters and Hyde; 1984). As in the case of the standard Rasch 
DIAMAP, a Rating Scale DIAMAP can give an immediate understanding of the 
measured trait for every respondent, based solely on their raw score! 




Figure 8: Boyd-Doig Mathematics Beliefs Inventory (Boyd and Doig; 1990). 

Figure 8 shows a Rating Scale DIAMAP for the Boyd-Doig Mathematics 
Beliefs Inventory (Boyd and Doig; 1990). This is a thirty-two item , four category, 
likert instrument assessing beliefs about mathematics and its teaching. The 
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thresholds for each category of response to a question is represented by a horizontal 
box of appropriate logit length, and the scaled raw scores form the horizontal axis. 
The actual response category of the respondent is ticked, while reading upwards 
from the respondent's raw score gives their most likely response to any item. In this 
way it is easy to identify aspects of belief which may be of most concern to the 
DIAMAP constructor. 

For example, in the DIAMAP shown in Figure 8, the response given to the first 
question indicates a belief that discovery methods of teaching may be frustrating to 
students. The raw score generated by the responses to all questions would indicate 
that this response is surprising for this respondent. Such surprising responses can be 
the basis for further investigation. A similar perspective can be taken of responses 
which appear to be more positive than expected, and be used as a foundation for 
building a better over all attitude. 

This brief outline of the coiistruction and use of DIAMAPs should serve to 
illustrate that good reporting methods can make Item Response Theory analyses 
more pertinent to the lay user and underscore the gains that can be made from 
using better analysis techniques. It is to be hoped that others will further develop 
the DIAMAP and thus improve the image of IRT specifically and good 
psychometrics in general. 
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